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This paper concentrates on a strategy that uses schoolwide data to identify secondary schools where students' 
academic achievement has been improving at a relatively fast rate, and other schools where it has not. Schools 
with the most involvement in vocational education can be selected from the group where achievement is 
increasing relatively fast, and from the group where it is not. These two sets of schools can then be compared in 
depth, to describe how vocational education is being used in each case. The in-depth study can supplement 
measures of academic achievement with other measures of student attainment, including work-related 
knowledge and skills. The results will answer two questions: 

• Do secondary schools where student academic achievement is improving at a relatively rapid rate 
organize and deliver vocational education differently than schools where student achievement is not 
improving as fast? 

• Do secondary schools where student academic achievement is improving at a relatively fast rate also 
have higher levels of student attainment in work-related knowledge and skill? 

Data for identifying schools could come from accountability systems that have been developed in various states 
to monitor school improvement. Another possible source of comparative schoolwide data would be networks 
such as the Southern Regional Education Board's High Schools That Work, or the U.S. Department of 
Education's New American High Schools. On-site observations and interviews would be necessary in most 
cases to describe how vocational education fits into the school's curriculum and organization, and to document 
teaching methods in vocational and nonvocational courses. This qualitative information would be used to learn 
what may be distinctive about vocational education in schools where achievement is increasing at a rapid rate. 
In addition, quantitative data can be used to estimate how changes in student achievement are related to prior 
changes in school practices, while controlling for demographic characteristics of students. 

This strategy for identifying promising practices in secondary vocational education has several advantages, 
which are explained in this paper: 

• The results, and the research process itself, can be directly useful in assisting improvement efforts of 
schools, districts, and states - as well as helping Congress identify effective uses of vocational education. 

• Starting with schools that have shown rapid improvement in student academic achievement is consistent 
with the emphasis in the 1998 Perkins Act on developing students' academic as well as vocational and 



technical skills, and integrating academic with vocational instruction. 

• Using change in student achievement, rather than the level of achievement, as the basis for selecting 
schools is consistent with the idea that schools can more fairly be held accountable for how fast students 
improve than for where they start. 

• The data to be used are more recent than the information in national longitudinal data sets. 

• Using schoolwide data can avoid statistical problems caused by certain kinds of selection bias. 

• Comparing schools is more feasible on a large scale than conducting studies that depend on random 
assignment of students to different treatments. 

Why Use Schoolwide Data? 

Some high schools with substantial involvement in vocational education are making rapid gains in student 
performance, while others are not. Why? One of the central questions for NAVE is to understand how 
vocational education is being used to advance students' academic and work-related achievement. 

To assess the effects of federally funded vocational education on the academic performance and labor market 
success of high school students, three main kinds of data are available. Advantages and disadvantages of each 
type of data are reviewed briefly here. The conclusion is that, given the constraints on NAVE's time and 
resources, the best way to gain insight into promising practices in high school vocational education is to start 
with available quantitative data on entire high schools, and select certain schools for in-depth qualitative study. 

Longitudinal data on individual students. First, longitudinal data on individual students from nationally 
representative surveys such as High School and Beyond (HSB), the National Educational Longitudinal Survey 
(NELS), and the National Longitudinal Survey of Youth (NLSY) can be used to estimate associations between 
certain aspects of students' high school experience and their subsequent success at school or work, controlling 
for students' background characteristics. The main advantage of these surveys is that they comprise large 
samples of students who represent the national population, and are therefore amenable to statistical analysis. 
Much of the published research on effects of vocational education has relied on these datasets. 

Several reviews of this research have been conducted (Bishop 1989; Boesel et al. 1994; Stem et al. 1995). They 
point to improvement in the research resulting from the availability of transcript data showing the actual 
courses taken by students while in high school. Using such data, a number of studies have found that high 
school vocational courses are more likely to lead to regular employment and high wages when graduates work 
in jobs where they can use the specific occupational knowledge and skills taught in those courses. On the other 
hand, students who take more vocational courses in high school are usually found to be less likely to enroll in 
postsecondary education. These findings are not very surprising. 

Another finding that emerges from several studies that use national longitudinal data sets is less obvious. High 
school students who combine a substantial academic curriculum with several vocational courses have been 
found to do better than students who omit either one of these two components. This was first noted by Kang 
and Bishop (1989), who discovered a positive interaction in the HSB data between the number of academic 
courses and the number of vocational courses in predicting post-high school earnings for males who did not 
attend college. Arum and Shavit (1995), also using the HSB data, used transcripts to identify students who had 
taken a set of advanced academic courses, a sequence of vocational courses, or both. Four years after senior 
year, individuals who had combined academic and vocational courses in high school had the greatest likelihood 
of being employed in professional, managerial, or skilled jobs, or being enrolled in postsecondary education. 
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Outcomes for this group were better, or at least as good, as those who had taken advanced academic courses 
but no vocational education. Levesque, Lauen, Teitelbaum, Alt, Librera, Joyner, and Nelson (1998), in addition 
to replicating these results with HSB data (tables 61.1 and 64), also found in the NELS data that students who 
combined a college-preparatory academic curriculum with a specific vocational sequence had gains in 
mathematics, reading, and science test scores during high school that were similar to the gains of students who 
took only the college-prep curriculum - and both of these groups gained substantially more than other students 
(tables 38 — 41). 

The problem with these findings based on longitudinal surveys is that they do not prove anything about cause 
and effect. The analysis can never fully eliminate the possibility of selection bias-that some unmeasured 
attributes of students account for both their participation in certain programs and their subsequent performance. 
For instance, the reason why students who are employed in jobs that use their specific occupational training 
earn higher wages may be that they enter high school already possessing specific abilities or interests which 
cause them to choose certain courses where they can excel, and also cause them to perform well on jobs that 
tap those same interests or abilities. The occupational courses themselves may contribute little or nothing to the 
abilities and interests that make these people valuable to employers. 

Similarly, the reason why students who take more vocational courses are less likely to enroll in postsecondary 
education may be that they come to high school less interested in the kind of book learning that schools 
emphasize, so they choose more hands-on courses while in high school, and are also more likeley to stop their 
formal schooling as soon as they graduate. The vocational courses themselves may do nothing to deter them 
from going on to college. Finally, the reason why students who combine a strong academic curriculum with an 
occupational sequence perform better both at school and at work may be that they start high school already 
possessing more ambition, energy, self-discipline, or awareness of what it takes to do well in the world, which 
causes them to take a doubly demanding curriculum in high school and also to excel in their post-high school 
pursuits. Their built-in get-up-and-go might enable them to succeed just as well without the extra coursework. 

Social scientists have tried hard to find ways of finessing the selection problem. One of the most sophisticated 
attempts is Altonji's 1995 analysis of data from the National Longitudinal Survey of the High School Class of 
1972, which he boldly describes as "the first systematic study of the effects of secondary school curriculum on 
postsecondary education and on success in the labor market" (p. 410). Altonji used the average number of 
courses in each subject taken by all students at a high school as an instrumental variable to predict the number 
of courses in that subject taken by each student at that high school. This was intended to reduce or eliminate the 
endogenous component of course-taking by individual students, which is what creates selection bias. He also 
compared the results with ordinary least-squares regressions. The results using instrumental variables often 
differed from those using ordinary least-squares, which may indicate that selection bias really is a problem. 

Surprisingly, foreign language courses stood out in Altonji's results as the best predictor of both post-high 
school wages and college attendance (tables 2 and 3). No other academic or vocational subject had a 
consistently significant association with wages. Altonji grouped vocational courses into "industrial" and 
"commercial"; both were negatively associated with college attendance. Altonji does not accept these results at 
face value (though the Modem Languages Association might be pleased if he did!). He surmises that foreign 
language courses may contribute to "general skill development," or else the result may be due to omitted 
variables or sampling error. He is puzzled by the small estimated effects of coursework in other subjects, 
particularly mathematics and science. Unfortunately, despite the attempt to rid the data of selection bias, the 
findings and discussion do not inspire much confidence in the possibility of reaching clear conclusions about 
cause and effect. 



A final problem with the national longitudinal data sets is that they captured high school experiences from the 
1970s (NLS72), 1980s (HSB and NLSY79), or early 1990s (NELS). The information therefore does not shed 
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any light on changes that may have occurred in high schools during the 1990s. The only new data set of this 
kind that might inform the current NAVE is a new National Longitudinal Survey of Youth that began in 1997 
(NLSY97). Unfortunately, the first wave of that survey did not collect transcript information, and the sample 
included a predominance of students in grades 9 and 10 who were too young to have had much opportunity to 
participate in vocational or career-related instruction. 

Comparing groups of students within schools. A second kind of data comes from comparing certain students 
in a school with other students in the same school. This can be done for more than one school at a time, if they 
offer comparable programs. Generally, schools are interested in knowing whether certain instructional practices 
lead to better performance by students. But simply comparing data for different students can be seriously 
misleading, again because of selection bias: students may be given certain kinds of instruction as a result of 
their level of performance. 

Consider a hypothetical example in which students who report that they make oral presentations in science 
class also obtain higher scores on a science achievement test. One explanation could be that high-achieving 
students are asked to make oral presentations because their teachers think they will set a good example for their 
classmates. Or an entire class in advanced science might be asked to do oral presentations because they have 
mastered the basics and the teacher believes they are ready for more individual work. In other words, being 
asked to do oral presentations could be the result, not the cause, of higher achievement. This is another 
example of selection bias. Consequently, the observed correlation does not imply that assigning more oral 
presentations for low-achieving students would necessarily improve their performance. 

It would be different if a school found that the average science score for students in the whole school improved 
after science teachers for some reason decided to begin assigning oral presentations to a larger proportion of 
students. This finding, based on change scores at the school level, would provide more compelling support for 
extension of the practice to additional students. This illustrates the benefit of using schoolwide data, as 
explained below. 

Examples of actual studies that have compared groups of students within schools include some evaluations of 
career academies (Stem, Dayton, and Raby 1998). A career academy is a small learning community in which a 
group of students take classes together from the same set of teachers for a period of two to four years. The 
curriculum combines a college-preparatory academic core with vocational courses related to the theme of the 
academy-for example, health careers, finance, or electronics. Partnerships with employers and other 
community groups provide internships and other connections between high school and the outside world. 

Three separate evaluations of career academies in California have compared the performance of academy 
students with similar students in the same school. That is, each academy student is individually matched with a 
nonacademy student of the same gender and race, whose prior academic record (grades, attendance, test scores, 
disciplinary infractions) is as similar as possible. These evaluations have followed both sets of students over 
time, and have consistently found that academy students attend school more regularly, earn higher grades, 
complete more courses, and are more likely to finish high school. However, it is possible that these results are 
affected by selection bias. If students who take the initiative of applying to a career academy have more of 
some unobserved positive characteristic such as ambition or planfulness, this could account for some or all of 
the difference in performance between the two groups. On the other hand, if academies tend to select students 
who are more troubled or recalcitrant, the observed results would underestimate the true effect of academies. 
The possibility of selection bias therefore makes the interpretation of results somewhat uncertain. 

A similar uncertainty occurs in evaluations which evaluate new forms of vocational education by comparing 
the performance of successive graduating classes of vocational "completers." If improvements in the vocational 
curriculum over time result in higher achieving students being attracted to vocational classes, this selection 
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effect could cause the measured performance of vocational completers to improve, even if there were no 
improvement in the average performance of students schoolwide. The best way to tell whether this is really 
happening would be to collect data for the school as a whole. 

The standard procedure to protect against selection bias is random assignment of students to the program and 
control groups. If students are randomly assigned, unobserved characteristics such as ambition or ability to plan 
or recalcitrance are unlikely to differ very much on average between the program and control groups. Random 
assignment has often been done in studies of educational practices at the level of the classroom, but it is more 
difficult to conduct a random assignment study of programs that span more than one classroom. Still, such 
studies have also been done despite the difficulty and expense. In particular, an ongoing random assignment 
study of career academies has found, among other things, that students in academies say they receive more 
support from teachers and peers, and find schoolwork more interesting and relevant, compared with 
nonacademy students (Kemple 1997). This evaluation also has found that career academy students participate 
more in career awareness activities and work-based learning. Among students whose prior performance in 
school was especially weak, academies also reduce the probability of dropping out. 

Even when random assignment can be done, however, the results may not be definitive. For example, suppose 
a high school starts a new program and invites students to apply. From among the applicant pool, some 
students are selected at random for the program, and the others become the control group. Suppose that both 
groups are doing well a few years later, but there is no significant difference between their results. A possible 
explanation could be that the students who applied for the program all had relatively high motivation to 
succeed, so they all did well whether they were admitted to the program or not. It is also possible that highly 
motivated students still need well-structured learning opportunities in order to succeed, and that students in the 
control group were able to find other such opportunities in the school. But if the number of well-structured 
learning opportunities available in the school is less than the number of potentially motivated students, and if 
the new program increased the number of opportunities, then eliminating the program would lead to poorer 
results, since some students in the program or in the control group would be unable to find good learning 
opportunities. This kind of effect could be detected only by using data for the school as a whole. 

Comparing rates of change between schools. The third option, collecting schoolwide data, would permit 
analysis at die school level. This is consistent with many current school reform efforts which emphasize the 
school site as the locus for planning, implementing, and monitoring improvements (e.g., Coalition of Essential 
Schools, HSTW, Obey-Porter programs). Most high schools are involved in a number of improvement efforts at 
any given time, with new initiatives happening constantly. In this environment, it is difficult to isolate the 
effect of any particular initiative because conditions for students not participating in that initiative are also 
likely to be changing. Nevertheless, it is possible for a school to learn from schoolwide data whether student 
performance is improving over time, and whether the rate of improvement is faster or slower than for other 
schools. Given that information, schools can attempt to explain why. The National Center for Research in 
Vocational Education (NCRVE) has engaged a number of high schools in a cycle of inquiry through which 
schools compare data in order to learn from their own and one another's experience. 2 Collecting and analyzing 
data on whole schools therefore can feed directly into the schools' own self-improvement efforts. This is an 
important advantage of such data. 

Using schoolwide data is consistent with the emphasis in the Perkins Act on developing students' academic as 
well as vocational and technical skills, and integrating academic with vocational instruction. Federal law in 
1990 and 1998 views vocational education as an integral part of the high school program. In fact, a review of 
transcripts for a national sample of 1994 high school graduates found that 97.2 percent of public high school 
students had taken at least one vocational course, and 90.8 percent had completed at least one specific 
occupational 2 course (Levesque et al. 1998, p. 12). The average high school graduate in 1994 had completed 
nearly four Carnegie units in vocational education courses (Levesque et al. 1998, p. 37). 
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Moreover, many students who take vocational courses go to college after graduating from high school. 
College-going students from the graduating class of 1982 accounted for 48 percent of all occupationally 
specific vocational coursework taken by members of that class during their four years of high school (Muraskin 
1993, p. 101). The number of students who combine vocational education with college-preparatory academic 
coursework has grown during the 1980s and 1990s. Among high school graduates who had taken three or more 
specific occupational courses, the proportion who also completed a course in chemistry rose from 15 percent in 
1982 to 34.6 percent in 1994, and the proportion who took physics grew from 7.8 to 13 percent (Levesque et al. 
1998, table 36.1). 

In the accountability provisions of the 1998 Perkins Act, Congress did not restrict states to measuring outcomes 
only for students participating in vocational education. Likewise, NAVE's studies of promising practices 
should not be limited to some subset of students who are defined as "vocational." Such a definition would have 
to be arbitrary and artificial, given the fact that almost all students take at least one vocational course. 
Identifying a group of high schoolers as "vocational students" would make no more sense than labeling some as 
"mathematics students," since virtually all students also take at least one mathematics class. It certainly would 
be possible to separate out students who took a relatively large number of vocational courses, just as it would 
be possible to distinguish those who took a lot of mathematics. But this kind of distinction is unnecessary and 
irrelevant if NAVE's purpose is to discover how vocational education is being used in conjunction with 
academic instruction to improve the performance of any or all students along a number of dimensions, 
including academic achievement and preparation for postsecondary education as well as work-related 
knowledge and skill. 

In deciding not to require states to identify a set of vocational students for purposes of accountability, Congress 
continued a strategy it adopted in the 1994 Improving America's Schools Act (IAS A). For nearly 30 years, Title 
I of the Elementary and Secondary Education Act had distributed federal money to low-income schools for 
compensatory education, with accountability provisions stipulating that the money could be spent only on 
students who were identified as meeting specific criteria. In addition to being cumbersome, the attempt to trace 
money to specific students led to those students being overtly labeled, segregated, and sometimes stigmatized. 
In 1994 this approach was replaced by IASA's new emphasis on schoolwide improvement. Although the 1998 
Perkins Act does not explicitly embrace the schoolwide strategy of IASA, it does avoid a definition of 
accountability that depends on identifying a subgroup of students. 

Given a focus on the whole school, another important principle is that schools should be compared in terms of 
change, not just the absolute level of student performance. Comparing schools in terms of absolute levels of 
performance can make some schools look bad just because their students come from low-income 
neighborhoods where educational performance tends to be lower than in more affluent places. Comparing 
change over time avoids this problem, creating a more level playing field— although shifts in student 
populations still may have to be taken into account in explaining why student achievement is improving faster 
in some schools than in others. 

Where to Find Schoolwide Data? 

The kind of schoolwide data needed would permit analysis of the trend over time in students' performance, and 
comparison with other schools. At a minimum, it would suffice to measure performance of a simple random 
sample of students at one or more grade levels, e.g., grade 12, and to repeat the same measurement from one 
year to the next. To ensure that changes are not entirely due to shifts in the composition of students, it would 
also be important to collect demographic data on the students whose performance is measured. Performance 
measures could include those specified in the accountability section of the Perkins Act: academic and technical 
skills attained, diplomas completed, postsecondary placements, and participation in nontraditional fields. 
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One likely source of this data would be individual states that have established accountability systems of their 
own. A number of states now collect data on individual schools over time, which permit comparisons among 
schools within a given state. According to Education Week (1999, pp. 85 — 87), the following 13 states 
maintained web sites that contain "report cards" on individual schools, with test data for more than one year in 
subjects that include English or mathematics at grade 1 1. In the following list, an asterisk denotes states where 
the tests include some performance measures in addition to multiple-choice tests: 

Alabama: English/language arts, history/social studies, mathematics, science 

Arizona: English/language arts, mathematics 

Florida: English/language arts, mathematics 

Georgia: English/language arts, history/social studies, mathematics, science 
Kentucky*: History/social studies, mathematics, science 
Michigan*: English/language arts, history/social studies, mathematics, science 
Nevada: English/language arts, mathematics 
New Jersey*: English/language arts, mathematics 

New York*: English/language arts, history/social studies, mathematics, science 
Ohic4: English/language arts, history/social studies, mathematics, science 
Pennsylvania*: English/language arts, mathematics 
Washington: English/language arts, mathematics 

West Virginia^: English/language arts, history/social studies, mathematics, science. 

Ten other states test 1 1th graders in subjects that include English or mathematics, but either do not include 
school report cards on their web sites or else do not report data for more than one year. These are: 

California: English/language arts, history/social studies, mathematics, science 

Idaho: English/language arts, mathematics 

Maine*: English/language arts, history/social studies, mathematics, science 
Mississippi: English/language arts, mathematics 
Missouri*: English/language arts, history/social studies 
Montana: English/language arts, history/social studies, mathematics, science 
Oklahoma: English/language arts, history/social studies, mathematics, science 






South Dakota: English/language arts, history/social studies, mathematics, science 
Utah: English/language arts, history/social studies, mathematics, science 
Wyoming*: English/language arts, mathematics. 



A particularly good source of data on the contribution of vocational education to school improvement is the 
multistate network called High Schools That Work (HSTW). This is the nation's first and largest effort to 
combine challenging academic courses and modem vocational educational studies to raise the achievement of 
high school students who were not enrolled in college-prep courses. As of 1999, the HSTW network includes 
close to 800 schools. The two major goals of the initiative are: (1) to raise the mathematics, science, 
communication, problem-solving and technical achievement of career-bound youth to the national average; and 
(2) to blend the essential content of traditional college preparatory studies -mathematics, science, language arts 
and social studies - with quality vocational and technical studies (Bottoms and Presson 1995; Southern 
Regional Education Board 1996). 

One of the key features of HSTW is the biennial assessment of student progress. Every two years, seniors who 
are classified as "vocational completers" take a special HSTW achievement test, based on the National 
Assessment of Educational Progress tests in mathematics, language, and science. Data from this test, along 
with results of a student survey, faculty survey, and follow-up of recent graduates, are compiled by the 
Educational Testing Service in a report to each school. Site visits from HSTW staff are occasions to examine 
school practices in the light of this data. The intent is to engender continuous improvement aided by periodic 
data collection. Unlike the data routinely collected by state accountability systems, the HSTW assessment 
includes information about vocational and academic courses taken by each student, and reports by students 
about specific instructional practices relating academic concepts to vocational applications. 

Beginning in 1998, some HSTW schools started collecting assessment data on schoolwide samples of students, 

instead of vocational completers only. Twenty-seven HSTW sites located in 20 different states^ collected 1998 
data from a fully representative sample of the senior class. Repeating this sampling procedure in the next 
assessment, scheduled for January-February 2000, would make it possible to compare schoolwide progress 
among these 27 high schools. Data would become available in the spring of 2000, making it possible to identify 
schools with relatively fast and slow rates of improvement in student achievement. On-site studies could then 
be conducted during the 2000-2001 school year, to determine how vocational education is being used in 
schools with rapid rates of improvement compared to other schools. This would allow time to analyze the 
qualitative and quantitative information and write up the results before the NAVE interim report is due at the 
end of 2001. 

The decision by these schools to repeat the schoolwide sampling procedure in 2000 will have to be made by 
August 1999, because it takes several months to get ready for the assessment. Schoolwide sampling in these 
schools entails an additional cost beyond what HSTW normally provides for collecting data on vocational 
completers. In 1998 the cost of testing the additional students was approximately $65,000, paid by NCRVE as 
part of its research study on using comparative data for school improvement. To ensure the continuation of this 
unique source of schoolwide data relating student achievement to vocational education, NAVE might have to 
provide all or part of the additional cost in 2000. 

Another possible source of data on schools that include vocational or work-related education as a central 
component of their curriculum is the set of New American High Schools identified by the Office of Vocational 
and Adult Education (OVAE) in the U.S. Department of Education. OVAE is asking these schools to provide 
trend data on one or more performance indicators in the following categories: 
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Academic performance: Advanced Placement and SAT scores, other achievement tests, percentage of 
graduating seniors meeting state university's entrance requirements, or percentage of senior 
portfolios/exhibitions/projects achieving satisfactory or higher rating; 

Technical or work-readiness skills: percentage of seniors in work-based learning who receive a 
satisfactory or higher rating by employer; 

High school graduation rate; 

Completion of a career program or major; 

Dropout rate: four-year completion rate, or annual dropout rate; 

Attendance: average days absent, or average daily attendance; 

Discipline problems: number of reported episodes, or number of suspensions/expulsions. 

It should be possible for the NAVE staff, assisted by the Advisory Panel and contractors, to determine fairly 
quickly which state accountability systems can provide useful data, along with HSTW and New American High 
Schools. There may also be other data sets that meet the desired criteria. Possibilities include the Beacon 
schools recognized by the Annenberg Foundation, schools that have won awards from the Association for 
Career and Technical Education, or schools participating in the International Skills Olympics for vocational 

students. 

How to Analyze Schoolwide Data? 

The main strategy proposed here is to conduct qualitative studies within a quantitative frame. That is, 
numerical data would be used to identify schools that show relatively rapid rates of improvement over time. 
Descriptive information from observations and interviews would then provide insights on how vocational 
education is delivered in these schools, compared to others. The result will be a demonstration of how 
vocational education can be part of an instructional package in schools where student attainment is growing 
relatively fast. 

Qualitative studies within a quantitative frame. The minimum necessary quantitative data for this analysis 
would consist of information on multiyear trends in student performance as measured by average scores on 
academic achievement tests. In addition to data on average performance (mean or median), it would be highly 
desirable also to measure performance in the high and low portions of the distribution (using quartile scores or 
standard deviations), so that schools can monitor trends across the whole spectrum of student achievement. 
Along with standardized tests of academic achievement, measures of work-related knowledge or skill, or at 
least application-oriented tests of academic achievement, would be especially pertinent to measuring the impact 
of vocational education^. In addition to test scores, other important indicators of student performance would 
include dropout and graduation rates, and postsecondary placements in work or further education. Follow-up 
data at least a year after graduation would also be revealing, but few schools or states collect such information. 

Given such indicators, analysis would begin by ranking schools according to how fast they are improving over 
time. The total number of schools to be identified for in-depth study will depend on the total budget available 
and the amount of qualitative detail desired. A plausible scenario would be to choose schools as follows: 

(1) Select three to five states where the accountability systems provide the necessary data on schools. In 
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each of these states, identify three to five schools where student achievement has improved at a rapid 
rate, and two or three schools where it has not. This would result in selection of between 15 and 40 
schools for direct, in-depth study. 

(2) From the 27 schools in HSTW with schoolwide data, select three to five schools with the fastest rates 
of growth in student achievement between 1998 and 2000. Also choose two or three where student 
achievement has not grown as rapidly. This would add between five and eight schools. 

(3) From the New American High Schools, choose another five to eight schools using the same 
procedure. 

These three sets add up to between 25 and 56 schools. If $5,000 were allocated to pay for visiting each of these 
schools and writing descriptions based on observations and interviews, the total cost would be between 
$125,000 and $280,000. 

Since the central purpose of this study is to understand the role of vocational education, it is necessary to 
ensure that vocational education has a substantial presence in each of the schools to be studied. Schools in the 
HSTW project may be presumed to satisfy this criterion, because vocational education plays an important part 
in the key practices promoted by this project. Similarly, attention to career-related instruction is one of the 
characteristics of the New American High Schools. 

However, in choosing schools based on data from state accountability systems, it will be necessary to verify 
involvement in vocational education before sending out teams to conduct in-depth studies. For instance, if the 
aim in a particular state is to select five schools where student achievement has grown rapidly and three where 
it has not, it may be necessary to start with a preliminary set of ten or fifteen fast-improving schools and six or 
nine slow-improving schools. From this preliminary set, schools for in-depth study can be chosen based on 
involvement in vocational education. Indicators of involvement in vocational education might include the 
average number of vocational courses students have taken, or the amount of money budgeted per pupil for 
vocational instruction. Information on vocational involvement may be obtained from the state education 
agency, local district offices, or preliminary telephone calls to the schools themselves. 

Once schools have been identified based on rate of improvement in student achievement and degree of 
involvement in vocational education, they can be visited by NAVE staff or contractors to conduct observations 
and interviews on the role of vocational education. The first question to be answered is whether and how 
schools where student academic achievement is improving at a relatively rapid rate organize and deliver 
vocational education differently than schools where student achievement is not improving as fast. Observations 
and interviews should look for differences along the following dimensions, among others: 

• Integrating academic and vocational curriculum and instruction. Do all or most students take a set of 
challenging academic classes as well as a coherent sequence of vocational courses? Do vocational 
courses reinforce academic skills and concepts? Do academic courses include work-related applications? 
Do students engage in projects that link academic and vocational subjects? 

• Work-based learning. Do all or most students engage in job shadowing, service learning, internships, 
school enterprises, or other forms of work-based learning? Are these experiences tied directly to 
academic courses? Do nonvocational teachers participate in work experiences outside of school, and do 
they supervise students who do the same? 

• Connecting secondary with postsecondary education. Do all or most students satisfy prerequisites for 
admission to a four-year college or university? Do they acquire occupational skills that will make it 
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easier for them to work their way through college? Are there explicit arrangements for students to earn 
college credits while in high school? 

In addition, a second important question is whether schools where student academic achievement is improving 
at a relatively fast rate also have higher levels of student attainment in work-related knowledge and skill. State 
accountability systems seldom if ever include measures of work-related knowledge and skill, so whatever data 
are available will have to be collected as part of the in-depth study of each school. Bishop (1999) has proposed 
a set of possible measures. Along with measures of students' actual knowledge and skill, it will also be useful 
to describe dimensions such as: 

• Career-related curriculum. Does the high school offer a set of curricular themes related to broad industry 
groupings or occupational clusters? Do all or most students choose such a theme? Do students share their 
core academic classes (English, mathematics, social studies, science) with others who have chosen the 
same theme? 

• Postsecondary pathways. Do curricular themes in high school lead directly to opportunities for 
employment and further education? Do students receive any kind of formal certification along the way? 

The core of the strategy proposed here would use descriptive information about these kinds of dimensions to 
compare schools where student performance is improving rapidly with schools where it is not. If schools with 
fast growth in student achievement organize and deliver vocational education differently than other schools, it 
will be possible to conclude at least that rapid gains in student attainment can occur when certain kinds of 
vocational education are practiced. The results will not prove that observed differences in programs or practices 
caused the differences in student performance, since it is conceivable that rapid growth in student performance 
is what permits these practices to be adopted. This can be debated. But it will still be useful to Congress and the 
field simply to know whether schools with rapid improvement are practicing a different kind of vocational 
education. 

Specific data collection procedures for the in-depth, on-site studies will not be detailed here, but there are 
several closely related precedents. Bradby and Teitelbaum (1998) piloted this kind of qualitative study within a 
quantitative frame for NCRVE, using data from HSTW. Rock and Ham (1999) describe a similar approach 
being used to evaluate Accelerated Schools. Hudis and Visher (1999) give descriptions of New American High 
Schools which show how qualitative accounts can compare certain practices among schools. Stasz (1999) 
outlines how qualitative studies of high schools can be conducted for NAVE. 

Purely quantitative studies. With aggegrated data from a few dozen schools, it is also possible to conduct 
purely quantitative analysis. Bloom (1999) has explained how to use trend data to test whether a particular 
intervention has had a significant impact on a set of schools. Under plausible assumptions, a set of 20 schools 
would be sufficient to detect an effect size of only 0.2 standard deviation. The virtue of this method is that it 
supports fairly strong inferences about cause and effect, based on changes in student performance after the 
intervention. However, Bloom’s procedure assumes that schools are adopting a well-defined new program or 
practice, beginning at a particular point in time. In most high schools, new initiatives are emerging all the time, 
so Bloom's method may not be appropriate. 

Nevertheless, aggregate data from a set of schools over time may be used to detect whether changes in school 
practices are associated with subsequent changes in student performance. Standard regression analysis can be 
applied here if the number of schools exceeds 30, and preferably more. Changes in student body characteristics 
can also be taken into account. Given NAVE’s two-year timeline, this kind of analysis would require 
retrospective data on school practices. 



Additional kinds of analysis can be conducted if data are available on individual students, including not only 
measures of their performance and background characteristics but also course-taking data and other indicators 
of their educational experience. For example, HSTW collect such information in their biennial assessment of 
seniors. Statistical techniques like hierarchical linear modeling can be used to measure the degree of 
association between students' individual educational histories and their performance on tests, as well as how 
those associations depend on school characteristics. For instance, the association between test scores and 
whether a student has taken a sequence of vocational courses may depend on whether most students in the 
school take such a vocational sequence, or only a few. NAVE may want to commission a special analysis of the 
unique HSTW data set to answer questions such as this. 



continue 
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